{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "DDJwQPZcupab",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# EECS 498-007/598-005 Assignment 5-1: Single-Stage Object Detector - YOLO\n",
    "\n",
    "Before we start, please put your name and UMID in following format\n",
    "\n",
    ": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "2KMxqLt1h2kx",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Your Answer:**   \n",
    "Firstname Lastname, #UMID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "BRIqwJUr2HuN",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Single-Stage Object Detector\n",
    "In this exercise you will implement a **single-stage** object detector, based on YOLO ([v1](https://arxiv.org/pdf/1506.02640.pdf) and [v2](https://arxiv.org/pdf/1612.08242.pdf)) and use it to train a model that can detect objects on novel images. We will also evaluate the detection accuracy using the classic metric mean Average Precision ([mAP](https://github.com/Cartucho/mAP)). In Part II of A5, you will implement a **two-stage** object detector, based on [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf). The main difference between the two is that single-stage detectors perform region proposal and classification simultaneously while two-stage detectors have them decoupled. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "LfBk3NtRgqaV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "MzqbYcKdz6ew",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup Code\n",
    "Before getting started, we need to run some boilerplate code to set up our environment, same as previous assignments. You'll need to rerun this setup code each time you start the notebook.\n",
    "\n",
    "First, run this cell load the autoreload extension. This allows us to edit .py source files, and re-import them into the notebook for a seamless editing and debugging experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 886,
     "status": "ok",
     "timestamp": 1604338879081,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "vxFGC0fKvkGC",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "atInOFC6vmCu",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Google Colab Setup\n",
    "Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n",
    "\n",
    "Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 26521,
     "status": "ok",
     "timestamp": 1604338904749,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "mlvjNCPxvmUE",
    "new_sheet": false,
    "outputId": "69397016-eca1-4243-9757-5401d25596d5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "OlF0J90Zvq7c",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['single_stage_detector_yolo.ipynb', 'two_stage_detector_faster_rcnn.ipynb', 'eecs598', 'single_stage_detector.py', 'two_stage_detector.py', 'a5_helper.py']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 26936,
     "status": "ok",
     "timestamp": 1604338905197,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "omkktd9ywDxp",
    "new_sheet": false,
    "outputId": "f1594c33-9422-4a1d-b209-50ab737ecc96",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# TODO: Fill in the Google Drive path where you uploaded the assignment\n",
    "# Example: If you create a 2020FA folder and put all the files under A5 folder, then '2020FA/A5'\n",
    "GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = None\n",
    "GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n",
    "print(os.listdir(GOOGLE_DRIVE_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Q56dbBs9wiIj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n",
    "\n",
    "```\n",
    "Hello from single_stage_detector.py!\n",
    "Hello from a5_helper.py!\n",
    "```\n",
    "\n",
    "as well as the last edit time for the file `single_stage_detector.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 34523,
     "status": "ok",
     "timestamp": 1604338912818,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "q0nvfShXwrRY",
    "new_sheet": false,
    "outputId": "32444d16-9c82-41e9-b233-41a6324b245b",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(GOOGLE_DRIVE_PATH)\n",
    "\n",
    "import time, os\n",
    "os.environ[\"TZ\"] = \"US/Eastern\"\n",
    "time.tzset()\n",
    "\n",
    "from single_stage_detector import hello_single_stage_detector\n",
    "hello_single_stage_detector()\n",
    "\n",
    "from a5_helper import hello_helper\n",
    "hello_helper()\n",
    "\n",
    "single_stage_detector_path = os.path.join(GOOGLE_DRIVE_PATH, 'single_stage_detector.py')\n",
    "single_stage_detector_edit_time = time.ctime(os.path.getmtime(single_stage_detector_path))\n",
    "print('single_stage_detector.py last edited on %s' % single_stage_detector_edit_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "_vm7Ign8x4b8",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Load several useful packages that are used in this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36661,
     "status": "ok",
     "timestamp": 1604338914983,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "HzRdJ3uhe1CR",
    "new_sheet": false,
    "outputId": "fa10c472-9574-41e5-9441-abf5406473f6",
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torchvision\n",
    "import statistics\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "import numpy as np\n",
    "import cv2\n",
    "import copy\n",
    "import shutil\n",
    "import os\n",
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from eecs598 import reset_seed, Solver\n",
    "from eecs598.grad import rel_error\n",
    "\n",
    "from a5_helper import *\n",
    "\n",
    "# for plotting\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['font.size'] = 16\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for mAP evaluation\n",
    "!rm -rf mAP\n",
    "!git clone https://github.com/Cartucho/mAP.git\n",
    "!rm -rf mAP/input/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "OvUDZWGU3VLV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will use GPUs to accelerate our computation in this notebook. Run the following to make sure GPUs are enabled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36634,
     "status": "ok",
     "timestamp": 1604338914984,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "RrAX9FOLpr9k",
    "new_sheet": false,
    "outputId": "2c374ea4-d4ae-4dcb-c855-7dc0d40580ba",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available:\n",
    "  print('Good to go!')\n",
    "else:\n",
    "  print('Please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "MjJ3uyYBg3Lw",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Load PASCAL VOC 2007 data\n",
    "During the majority of our homework assignments so far, we have used the CIFAR-10 dataset for image classification tasks.\n",
    "\n",
    "We will need to use a new dataset for object detection. In order to train and evaluate object detection models, we need a dataset where each image is annotated with a *set* of *bounding boxes*, where each box gives the category label and spatial extent of some object in the image.\n",
    "\n",
    "We will use the [PASCAL VOC 2007](http://host.robots.ox.ac.uk/pascal/VOC/) dataset, which provides annotations of this form. PASCAL VOC ran a series of yearly computer vision competitions from 2005 to 2012, predating the ImageNet challenge which we have discussed in class.\n",
    "\n",
    "The data from the 2007 challenge used to be one of the most popular datasets for evaluating object detection. It is much smaller than more recent object detection datasets such as [COCO](http://cocodataset.org/#home), and thus easier to manage in an homework assignment.\n",
    "\n",
    "The function `get_pascal_voc2007_data` in `a5_helper.py` will download the PASCAL VOC 2007 dataset and return it as a PyTorch Dataset object. Run the following cell to download the training and validation sets for the PASCAL VOC 2007 dataset.\n",
    "\n",
    "The [`Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) objects returned from the above function returns annotations for each image as a nested set of dictionary objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 885,
     "referenced_widgets": [
      "72cee37f094d4bf98286cf609591a065",
      "af39fda1643342619373531854ec5a6a",
      "fcb7f0f5206242f9abef713c3fcf3d59",
      "c6de2c2b7f6d43f3bc5db37780aa6bc9",
      "39f2c90a30784ccd95c885861a8ae353",
      "7698f88da6ed494886f88d808869555c",
      "ad0732df20c546ad99c079fb55fd5110",
      "98ee12dbedf24de884be7c596bf4228a"
     ]
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 99807,
     "status": "ok",
     "timestamp": 1604338978191,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "MmEP5KQJzk0d",
    "new_sheet": false,
    "outputId": "86f1c3a7-925d-4df4-c855-9d50ded95ec9",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# uncomment below to use the mirror link if the original link is broken and move it to GOOGLE_DRIVE_PATH\n",
    "# !wget https://web.eecs.umich.edu/~justincj/data/VOCtrainval_06-Nov-2007.tar\n",
    "# shutil.move('VOCtrainval_06-Nov-2007.tar', os.path.join(GOOGLE_DRIVE_PATH, 'VOCtrainval_06-Nov-2007.tar'))  \n",
    "train_dataset = get_pascal_voc2007_data(GOOGLE_DRIVE_PATH, 'train')\n",
    "val_dataset = get_pascal_voc2007_data(GOOGLE_DRIVE_PATH, 'val')\n",
    "\n",
    "# an example on the raw annotation\n",
    "print(json.dumps(train_dataset[1][1]['annotation'], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "J5MjBX9bkBtA",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In order to use these annotations to train our model, we need to convert this nested dictionary data structure into a set of PyTorch tensors.\n",
    "\n",
    "We also need to preprocess the image, converting it to a PyTorch tensor and resizing it to 224x224. Real object detection systems typically work with much higher-resolution images, but we will use a low resolution for computational efficiency in this assignment.\n",
    "\n",
    "We also want to train our models using minibatches of data, so we need to group the annotations from several images into minibatches.\n",
    "\n",
    "We perform both of these functions by using a customized PyTorch [DataLoader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) object, which we have written for you. Those functions are all available in `a5_helper.py`. \n",
    "\n",
    "Training with the entire PASCAL VOC will be too computationally expensive for this homework assignment, so we can subsample the dataset by wrapping each `Dataset` object in a [`Subset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset) object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 99806,
     "status": "ok",
     "timestamp": 1604338978192,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "XL-7Em_A1kdS",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_dataset = torch.utils.data.Subset(train_dataset, torch.arange(0, 2500)) # use 2500 samples for training\n",
    "train_loader = pascal_voc2007_loader(train_dataset, 10)\n",
    "val_loader = pascal_voc2007_loader(val_dataset, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "HTyRHqwlC1Au",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The `DataLoader` objects return batches of data.\n",
    "\n",
    "The first output from the `DataLoader` is a Tensor `img` of shape `(B, 3, 224, 224)`. This is a batch of `B` images, similar to what we have seen in classification datasets.\n",
    "\n",
    "The second output from the `DataLoader` is a Tensor `ann` of shape `(B, N, 5)` giving information about all objects in all images of the batch. `ann[i, j] = (x_tl, y_tl, x_br, y_br, class)` gives information about the `j`th object in `img[i]`. The position of the top-left corner of the box is `(x_tl, y_tl)` and the position of the bottom-right corner of the box is `(x_br, y_br)`. These positions are in the coordinate system of the original image (before it was resized to 224 x 224). `class` is an integer giving the category label for this bounding box.\n",
    "\n",
    "Each image can have different numbers of objects. If `img[i]` has $N_i$ objects, then $N=\\max_i N_i$ is the maximum number of objects per image among all objects in the batch; this value can vary from batch to batch. For the images that have fewer than $N$ annotated objects, only the first $N_i$ rows of $anns[i]$ contain annotations; the remaining rows are padded with -1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 110867,
     "status": "ok",
     "timestamp": 1604338989282,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "nZVYFJD32I_l",
    "new_sheet": false,
    "outputId": "57eeeb01-b02e-4e50-ccad-d180bcd9d382",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_loader_iter = iter(train_loader)\n",
    "img, ann, _, _, _ = train_loader_iter.next()\n",
    "\n",
    "print('img has shape: ', img.shape)\n",
    "print('ann has shape: ', ann.shape)\n",
    "\n",
    "print('Image 1 has only two annotated objects, so ann[1] is padded with -1:')\n",
    "print(ann[1])\n",
    "\n",
    "print('\\nImage 2 has six annotated objects:, so ann[2] is not padded:')\n",
    "print(ann[2])\n",
    "\n",
    "print('\\nEach row in the annotation tensor indicates (x_tl, y_tl, x_br, y_br, class).')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "GqISg-cs6vKM",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Coordinate transformation\n",
    "It's a good practice to use a consistent coordinate system for all the spatial-related computations (e.g., anchors, proposals). **In this assignment, we use the coordinate system defined by the CNN activation map (of shape 7x7), where the top-left corner is (0, 0) and the bottom-right corner is (7, 7). The horizontal axis is the x axis and the vertical axis is the y axis.**\n",
    "\n",
    "The following function defines the transformation from the original image coordinate system (pixels, and the top-left corner is (0, 0)) to the activation map coordinate system and vice versa. You can find the function in `a5_helper.py`\n",
    "\n",
    "```python\n",
    "def coord_trans(bbox, w_pixel, h_pixel, w_amap=7, h_amap=7, mode='a2p'):\n",
    "  \"\"\"\n",
    "  Coordinate transformation function. It converts the box coordinate from\n",
    "  the image coordinate system to the activation map coordinate system and vice versa.\n",
    "  In our case, the input image will have a few hundred of pixels in\n",
    "  width/height while the activation map is of size 7x7.\n",
    "\n",
    "  Input:\n",
    "  - bbox: Could be either bbox, anchor, or proposal, of shape Bx*x4\n",
    "  - w_pixel: Number of pixels in the width side of the original image, of shape B\n",
    "  - h_pixel: Number of pixels in the height side of the original image, of shape B\n",
    "  - w_amap: Number of pixels in the width side of the activation map, scalar\n",
    "  - h_amap: Number of pixels in the height side of the activation map, scalar\n",
    "  - mode: Whether transfer from the original image to activation map ('p2a') or\n",
    "          the opposite ('a2p')\n",
    "\n",
    "  Output:\n",
    "  - resized_bbox: Resized box coordinates, of the same shape as the input bbox\n",
    "  \"\"\"\n",
    "\n",
    "  assert mode in ('p2a', 'a2p'), 'invalid coordinate transformation mode!'\n",
    "  assert bbox.shape[-1] >= 4, 'the transformation is applied to the first 4 values of dim -1'\n",
    "  \n",
    "  if bbox.shape[0] == 0: # corner cases\n",
    "    return bbox\n",
    "\n",
    "  resized_bbox = bbox.clone()\n",
    "  # could still work if the first dim of bbox is not batch size\n",
    "  # in that case, w_pixel and h_pixel will be scalars\n",
    "  resized_bbox = resized_bbox.view(bbox.shape[0], -1, bbox.shape[-1])\n",
    "  invalid_bbox_mask = (resized_bbox == -1) # indicating invalid bbox\n",
    "\n",
    "  if mode == 'p2a':\n",
    "    # pixel to activation\n",
    "    width_ratio = w_pixel * 1. / w_amap\n",
    "    height_ratio = h_pixel * 1. / h_amap\n",
    "    resized_bbox[:, :, [0, 2]] /= width_ratio.view(-1, 1, 1)\n",
    "    resized_bbox[:, :, [1, 3]] /= height_ratio.view(-1, 1, 1)\n",
    "  else:\n",
    "    # activation to pixel\n",
    "    width_ratio = w_pixel * 1. / w_amap\n",
    "    height_ratio = h_pixel * 1. / h_amap\n",
    "    resized_bbox[:, :, [0, 2]] *= width_ratio.view(-1, 1, 1)\n",
    "    resized_bbox[:, :, [1, 3]] *= height_ratio.view(-1, 1, 1)\n",
    "\n",
    "  resized_bbox.masked_fill_(invalid_bbox_mask, -1)\n",
    "  resized_bbox.resize_as_(bbox)\n",
    "  return resized_bbox\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "Notes: All the coordinates are in float precision. In later sections, we use the activation map coordinate system for all computations except for visualization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "48Cocwotg9Ly",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Detection Visualizer\n",
    "\n",
    "We also use a function to visualize our detections. You can find this function in `eecs598/vis.py`. We have included it below for your convenience\n",
    "\n",
    "```python\n",
    "def detection_visualizer(img, idx_to_class, bbox=None, pred=None):\n",
    "    \"\"\"\n",
    "    Data visualizer on the original image. \n",
    "    Support both GT box input and proposal input.\n",
    "    \n",
    "    Input:\n",
    "    - img: PIL Image input\n",
    "    - idx_to_class: Mapping from the index (0-19) to the class name\n",
    "    - bbox: GT bbox (in red, optional), a tensor of shape Nx5, where N is\n",
    "            the number of GT boxes, 5 indicates (x_tl, y_tl, x_br, y_br, class)\n",
    "    - pred: Predicted bbox (in green, optional), a tensor of shape N'x6, where\n",
    "            N' is the number of predicted boxes, 6 indicates\n",
    "            (x_tl, y_tl, x_br, y_br, class, object confidence score)\n",
    "    \"\"\"\n",
    "\n",
    "    img_copy = np.array(img).astype('uint8')\n",
    "\n",
    "    if bbox is not None:\n",
    "        for bbox_idx in range(bbox.shape[0]):\n",
    "            one_bbox = bbox[bbox_idx][:4]\n",
    "            cv2.rectangle(img_copy, (one_bbox[0], one_bbox[1]), (one_bbox[2],\n",
    "                        one_bbox[3]), (255, 0, 0), 2)\n",
    "            if bbox.shape[1] > 4: # if class info provided\n",
    "                obj_cls = idx_to_class[bbox[bbox_idx][4].item()]\n",
    "                cv2.putText(img_copy, '%s' % (obj_cls),\n",
    "                          (one_bbox[0], one_bbox[1]+15),\n",
    "                          cv2.FONT_HERSHEY_PLAIN, 1.0, (0, 0, 255), thickness=1)\n",
    "\n",
    "    if pred is not None:\n",
    "        for bbox_idx in range(pred.shape[0]):\n",
    "            one_bbox = pred[bbox_idx][:4]\n",
    "            cv2.rectangle(img_copy, (one_bbox[0], one_bbox[1]), (one_bbox[2],\n",
    "                        one_bbox[3]), (0, 255, 0), 2)\n",
    "            \n",
    "            if pred.shape[1] > 4: # if class and conf score info provided\n",
    "                obj_cls = idx_to_class[pred[bbox_idx][4].item()]\n",
    "                conf_score = pred[bbox_idx][5].item()\n",
    "                cv2.putText(img_copy, '%s, %.2f' % (obj_cls, conf_score),\n",
    "                            (one_bbox[0], one_bbox[1]+15),\n",
    "                            cv2.FONT_HERSHEY_PLAIN, 1.0, (0, 0, 255), thickness=1)\n",
    "\n",
    "    plt.imshow(img_copy)\n",
    "    plt.axis('off')\n",
    "    plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Fnej4xbT74Ka",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Before we move on, we import several key functions in the box below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 110866,
     "status": "ok",
     "timestamp": 1604338989283,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "PLnNVxSf77uT",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from eecs598.vis import detection_visualizer\n",
    "from a5_helper import coord_trans\n",
    "\n",
    "# Import Pascal clas mappings\n",
    "from a5_helper import class_to_idx, idx_to_class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "X4WmocEyiXWa",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Visualize PASCAL VOC 2007\n",
    "It is always good practice to try and visualize parts of your dataset before you build a model.\n",
    "\n",
    "Here we sample some  images from the PASCAL VOC 2007 training set, and visualize the ground-truth object boxes and category labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 111357,
     "status": "ok",
     "timestamp": 1604338989775,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "ld1s28Z4fyL5",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# default examples for visualization\n",
    "reset_seed(0)\n",
    "batch_size = 3\n",
    "sampled_idx = torch.linspace(0, len(train_dataset)-1, steps=batch_size).long()\n",
    "\n",
    "# get the size of each image first\n",
    "h_list = []\n",
    "w_list = []\n",
    "img_list = [] # list of images\n",
    "MAX_NUM_BBOX = 40\n",
    "box_list = torch.LongTensor(batch_size, MAX_NUM_BBOX, 5).fill_(-1) # PADDED GT boxes\n",
    "\n",
    "for idx, i in enumerate(sampled_idx):\n",
    "  # hack to get the original image so we don't have to load from local again...\n",
    "  img, ann = train_dataset.__getitem__(i)\n",
    "  img_list.append(img)\n",
    "\n",
    "  all_bbox = ann['annotation']['object']\n",
    "  if type(all_bbox) == dict:\n",
    "    all_bbox = [all_bbox]\n",
    "  for bbox_idx, one_bbox in enumerate(all_bbox):\n",
    "    bbox = one_bbox['bndbox']\n",
    "    obj_cls = one_bbox['name']\n",
    "    box_list[idx][bbox_idx] = torch.LongTensor([int(bbox['xmin']), int(bbox['ymin']),\n",
    "      int(bbox['xmax']), int(bbox['ymax']), class_to_idx[obj_cls]])\n",
    "\n",
    "  # get sizes\n",
    "  img = np.array(img)\n",
    "  w_list.append(img.shape[1])\n",
    "  h_list.append(img.shape[0])\n",
    "\n",
    "w_list = torch.as_tensor(w_list).float().cuda()\n",
    "h_list = torch.as_tensor(h_list).float().cuda()\n",
    "box_list = torch.as_tensor(box_list).float().cuda()\n",
    "resized_box_list = coord_trans(box_list, w_list, h_list, mode='p2a') # on activation map coordinate system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 112416,
     "status": "ok",
     "timestamp": 1604338990852,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "v04D-gwEiWqY",
    "new_sheet": false,
    "outputId": "188a35f2-4ece-4620-aad0-b6afb73811c1",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize GT boxes\n",
    "for i in range(len(img_list)):\n",
    "  valid_box = sum([1 if j != -1 else 0 for j in box_list[i][:, 0]])\n",
    "  detection_visualizer(img_list[i], idx_to_class, box_list[i][:valid_box])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "IAa1Kvl2P_2k",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Detector Backbone Network\n",
    "Here, we use [MobileNet v2](https://pytorch.org/hub/pytorch_vision_mobilenet_v2/) for image feature extraction. The code is based on Assignment 4-2. The function is included in a5_helper and can be seen below. \n",
    "\n",
    "```python\n",
    "class FeatureExtractor(nn.Module):\n",
    "  \"\"\"\n",
    "  Image feature extraction with MobileNet.\n",
    "  \"\"\"\n",
    "  def __init__(self, reshape_size=224, pooling=False, verbose=False):\n",
    "    super().__init__()\n",
    "\n",
    "    self.mobilenet = models.mobilenet_v2(pretrained=True)\n",
    "    self.mobilenet = nn.Sequential(*list(self.mobilenet.children())[:-1]) \n",
    "\n",
    "    # average pooling\n",
    "    if pooling:\n",
    "      self.mobilenet.add_module('LastAvgPool', nn.AvgPool2d(math.ceil(reshape_size/32.))) # input: N x 1280 x 7 x 7\n",
    "\n",
    "    for i in self.mobilenet.named_parameters():\n",
    "      i[1].requires_grad = True # fine-tune all parameters\n",
    "\n",
    "    if verbose:\n",
    "      summary(self.mobilenet.cuda(), (3, reshape_size, reshape_size))\n",
    "  \n",
    "  def forward(self, img, verbose=False):\n",
    "    \"\"\"\n",
    "    Inputs:\n",
    "    - img: Batch of resized images, of shape Nx3x224x224\n",
    "    \n",
    "    Outputs:\n",
    "    - feat: Image feature, of shape Nx1280 (pooled) or Nx1280x7x7\n",
    "    \"\"\"\n",
    "    num_img = img.shape[0]\n",
    "    \n",
    "    img_prepro = img\n",
    "\n",
    "    feat = []\n",
    "    process_batch = 500\n",
    "    for b in range(math.ceil(num_img/process_batch)):\n",
    "      feat.append(self.mobilenet(img_prepro[b*process_batch:(b+1)*process_batch]\n",
    "                              ).squeeze(-1).squeeze(-1)) # forward and squeeze\n",
    "    feat = torch.cat(feat)\n",
    "    \n",
    "    if verbose:\n",
    "      print('Output feature shape: ', feat.shape)\n",
    "    \n",
    "    return feat\n",
    "```\n",
    "\n",
    "Now, let's see what's inside MobileNet v2. Assume we have a 3x224x224 image input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "bbdfd4e4a86a48fb86b2063510dd53d9",
      "876bc727f1a341868e3ef42ba2763408",
      "b7417b5be2ee4915be4dc010d170ce7c",
      "69460e1899694f06a47c55f4fd2cab38",
      "f45f1f174a9e479aa6f6e1fa61527af1",
      "9afd2271777144eea2b996de9747975c",
      "8d6a6ba882824d40a258d0eca1b70335",
      "cb76667a3a5f4e9a8100e2407b2bdae8"
     ]
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 112820,
     "status": "ok",
     "timestamp": 1604338991274,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "_pV0Lau_yDwX",
    "new_sheet": false,
    "outputId": "d44e09cf-d937-4449-9be6-906de55edf14",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "model = FeatureExtractor(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "zz32q025joxy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Anchor and Proposal\n",
    "The concept of *anchor* is introduced in [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf) and later used in numerous later works including [YOLO v2](https://arxiv.org/pdf/1612.08242.pdf). The definition of anchor from the original paper is summarized as follows:\n",
    "\n",
    "After passing the input image through the backbone network, we have a convolutional feature map of shape $(C, 7, 7)$ which we interpret as a 7x7 grid of $C$-dimensional features. At each point in this grid, we consider a set of $A$ *anchor boxes* of different sizes and shapes; for each anchor box we classify it as either an object or background box. The total number of anchor boxes that we consider for the entire input image is $(A, 7, 7)$; we predict classification scores of this shape by applying a sequence of convolutional layers to the backbone features.\n",
    "\n",
    "We slide a small network (e.g., 3x3 conv layer) over the CNN activation feature map. We call this 3x3 conv a sliding window. At each sliding-window location (i.e., centered at each position of the 7x7 activation cell), we simultaneously predict multiple region *proposals*, where the number of proposals for each location is denoted as $A=9$.\n",
    "\n",
    "Later, we will have an object proposal layer outputs\n",
    "A-D scores that estimate probability of object for each proposal, a bounding box regression layer to produce 4A-D outputs encoding\n",
    "the coordinates of A boxes, and a region classification layer to produce 20-D outputs indicating the probability of being each object category (shared by all A anchors). The A *proposals* are parameterized relative to A reference boxes, which we call *anchors*. An anchor is centered at the sliding window in question, and is associated with a shape (e.g., 1x1, 3x3, 5x5). The list of anchor shapes are provided next.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "etBYc7rbj35F",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Anchor shapes\n",
    "At each spatial position of the 7x7 backbone features, we consider a set of $A$ anchor boxes. Different spatial positions all use anchors of the same shape.\n",
    "\n",
    "The shapes of the anchor boxes are a hyperparameter. We will provide the anchor shapes for you, and we will re-use the same anchor shapes for both A5-1 and A5-2. In some papers (e.g. YOLO v2), the anchor shapes are determined in a data-driven way by clustering the set of ground-truth box sizes, but for simplicity we will not use that approach in this assignment.\n",
    "\n",
    "Note that anchors could be much larger than the 3x3 sliding window (e.g., 5x5) since the receptive field of activation cell on the original image could be large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 112803,
     "status": "ok",
     "timestamp": 1604338991274,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "O5w-EUJekJj-",
    "new_sheet": false,
    "outputId": "b343b436-9946-44ef-90af-8d8ccdc6f348",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Declare variables for anchor priors, a Ax2 Tensor where A is the number of anchors.\n",
    "# Hand-picked, same as our two-stage detector.\n",
    "anchor_list = torch.tensor([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [2, 3], [3, 2], [3, 5], [5, 3]]).float().cuda()\n",
    "print(anchor_list.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "uochvAlqkgr8",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Activation Grid Generator\n",
    "In order to place anchors centered at each position of the 7x7 grid of backbone features, we need to know the spatial position of the center of each cell in the 7x7 grid of features.\n",
    "\n",
    "This function will compute these center coordinates for us.\n",
    "\n",
    "```python\n",
    "def GenerateGrid(batch_size, w_amap=7, h_amap=7, dtype=torch.float32, device='cuda'):\n",
    "  \"\"\"\n",
    "  Return a grid cell given a batch size (center coordinates).\n",
    "\n",
    "  Inputs:\n",
    "  - batch_size, B\n",
    "  - w_amap: or W', width of the activation map (number of grids in the horizontal dimension)\n",
    "  - h_amap: or H', height of the activation map (number of grids in the vertical dimension)\n",
    "  - W' and H' are always 7 in our case while w and h might vary.\n",
    "  \n",
    "  Outputs:\n",
    "  grid: A float32 tensor of shape (B, H', W', 2) giving the (x, y) coordinates\n",
    "        of the centers of each feature for a feature map of shape (B, D, H', W')\n",
    "  \"\"\"\n",
    "  w_range = torch.arange(0, w_amap, dtype=dtype, device=device) + 0.5\n",
    "  h_range = torch.arange(0, h_amap, dtype=dtype, device=device) + 0.5\n",
    "\n",
    "  w_grid_idx = w_range.unsqueeze(0).repeat(h_amap, 1)\n",
    "  h_grid_idx = h_range.unsqueeze(1).repeat(1, w_amap)\n",
    "  grid = torch.stack([w_grid_idx, h_grid_idx], dim=-1)\n",
    "  grid = grid.unsqueeze(0).repeat(batch_size, 1, 1, 1)\n",
    "\n",
    "  return grid\n",
    "```\n",
    "\n",
    "We now visualize its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 113824,
     "status": "ok",
     "timestamp": 1604338992312,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "UCHpDXV0sdEX",
    "new_sheet": false,
    "outputId": "7a54900f-d3d6-4731-f4a6-d70db9415873",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualization\n",
    "# simply create an activation grid where the cells are in green and the centers in red\n",
    "# you should see the entire image divided by a 7x7 grid, with no gaps on the edges\n",
    "\n",
    "grid_list = GenerateGrid(w_list.shape[0])\n",
    "\n",
    "center = torch.cat((grid_list, grid_list), dim=-1)\n",
    "grid_cell = center.clone()\n",
    "grid_cell[:, :, :, [0, 1]] -= 1. / 2.\n",
    "grid_cell[:, :, :, [2, 3]] += 1. / 2.\n",
    "center = coord_trans(center, w_list, h_list)\n",
    "grid_cell = coord_trans(grid_cell, w_list, h_list)\n",
    "\n",
    "for img, anc, grid in zip(img_list, center, grid_cell):\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4), grid.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "R0knLi5KkxoS",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Anchor Generator\n",
    "At this point we have defined the spatial sizes of anchors that we consider at each grid point, and we have computed the center of each grid point. We now combine these two pieces of information to compute the positions of all anchor boxes for the entire image.\n",
    "\n",
    "\n",
    "We denote the anchor coordinates as ($x_{tl}^a$, $y_{tl}^a$, $x_{br}^a$, $y_{br}^a$), indicating the coordinates of the top-left corner and the bottom-right corner accordingly. The following function returns all the anchors given the anchor shapes and the grid cell. **Note that the center of an anchor overlaps a grid cell center.**\n",
    "\n",
    "Run the following to check your implementation. You should see errors on the order of 1e-8 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 114781,
     "status": "ok",
     "timestamp": 1604338993286,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "gffaPg4Dsfux",
    "new_sheet": false,
    "outputId": "a661c711-b32f-4187-e7d7-d2b835c7caa5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import GenerateAnchor\n",
    "\n",
    "# sanity check\n",
    "anc_list = GenerateAnchor(anchor_list, grid_list)\n",
    "assert anc_list.shape == torch.Size([3, 9, 7, 7, 4]), 'shape mismatch!'\n",
    "\n",
    "expected_anc_list_mean = torch.tensor([[-1.25000000, -0.87500000,  2.25000000,  1.87500000],\n",
    "                                       [ 1.75000000, -0.87500000,  5.25000000,  1.87500000],\n",
    "                                       [ 4.75000000, -0.87500000,  8.25000000,  1.87500000],\n",
    "                                       [-1.25000000,  1.12500000,  2.25000000,  3.87500000],\n",
    "                                       [ 1.75000000,  1.12500000,  5.25000000,  3.87500000],\n",
    "                                       [ 4.75000000,  1.12500000,  8.25000000,  3.87500000],\n",
    "                                       [-1.25000000,  3.12500000,  2.25000000,  5.87500000],\n",
    "                                       [ 1.75000000,  3.12500000,  5.25000000,  5.87500000],\n",
    "                                       [ 4.75000000,  3.12500000,  8.25000000,  5.87500000],\n",
    "                                       [-1.25000000,  5.12500000,  2.25000000,  7.87500000],\n",
    "                                       [ 1.75000000,  5.12500000,  5.25000000,  7.87500000],\n",
    "                                       [ 4.75000000,  5.12500000,  8.25000000,  7.87500000]], dtype=torch.float32, device='cuda')\n",
    "print('rel error: ', rel_error(expected_anc_list_mean, anc_list[0, [1,3,6,8], ::2, ::3, :].view(-1, 12, 4).mean(0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 115917,
     "status": "ok",
     "timestamp": 1604338994440,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "xbxTvazYyV7P",
    "new_sheet": false,
    "outputId": "8cf427f9-9241-45c1-c867-5567c5d0c5b3",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualization\n",
    "print('*'*80)\n",
    "print('All nine anchors should be exactly centered:')\n",
    "anc_list = GenerateAnchor(anchor_list, grid_list[:, 3:4, 3:4])\n",
    "for img, anc in zip(img_list, coord_trans(anc_list, w_list, h_list)):\n",
    "  print(anc.shape)\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4))\n",
    "\n",
    "print('*'*80)\n",
    "print('All anchors of the image (cluttered):')\n",
    "anc_list = GenerateAnchor(anchor_list, grid_list) # all\n",
    "for img, anc in zip(img_list, coord_trans(anc_list, w_list, h_list)):\n",
    "  print(anc.shape)\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "KCzcNX3Zsi3n",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Proposal Generator\n",
    "If we only use anchors to propose object locations, we can only cover 9x7x7=441 regions in the image. What if an object does not fall into any of the regions?\n",
    "\n",
    "Hence, in the recent literatures (e.g., [Fast R-CNN](https://arxiv.org/pdf/1504.08083.pdf)), the detector predicts transformations that convert anchor boxes into *region proposals*.\n",
    "\n",
    "So far, we have represented anchors boxes by the coordinates of their top-left and bottom-right corners $(x_{tl}^a, y_{tl}^a, x_{br}^a, y_{br}^a)$. When converting anchors to proposals, it will be more convenient to parameterize boxes by the xy coordinate of their center, and their with and height: $(x_c^a,y_c^a,w^a,h^a)$.\n",
    "\n",
    "Now, consider an anchor box with center, width and height $(x_c^a,y_c^a,w^a,h^a)$.\n",
    "The network will predict a *transformation* $(t^x, t^y, t^w, t^h)$; applying this transformation to the anchor yields a *region proposal* with center, width and height $(x_c^p,y_c^p,w^p,h^p)$.\n",
    "YOLO and Faster R-CNN use slightly different formulas to convert anchors into proposals. Here you need to implement both formulations.\n",
    "\n",
    "### YOLO\n",
    "For YOLO, we assume that $t^x$ and $t^y$ are both in the range $-0.5\\leq t^x,t^y\\leq 0.5$, while $t^w$ and $t^h$ are real numbers in the range $(-\\infty, \\infty)$. Then we have:\n",
    "- $x_c^p = x_c^a + t^x$\n",
    "- $y_c^p = y_c^a + t^y$\n",
    "- $w^p = w_a exp(t^w)$\n",
    "- $h^p = h_a exp(t^h)$\n",
    "\n",
    "### Faster R-CNN\n",
    "For Faster R-CNN, we assume that all transformation parameters $t^x, t^y, t^w, t_h$ are real numbers in the range $(-\\infty, \\infty)$. Then we have:\n",
    "- $x_c^p = x_c^a + t^xw^a$\n",
    "- $y_c^p = y_c^p + t^yh^a$\n",
    "- $w^p = w_a exp(t^w)$\n",
    "- $h^p = h_a exp(t^h)$\n",
    "\n",
    "\n",
    "### Training\n",
    "During training, we compute the ground-truth transformation $(\\hat{t^x}, \\hat{t^y}, \\hat{t^w}, \\hat{t^h})$ that would transform the anchor box $(x_c^a,y_c^a,w^a,h^a)$ into the the ground-truth box $(x_c^{gt},y_c^{gt},w^{gt},h^{gt})$. We then apply a regression loss that penalizes differences between the predicted transform $(t^x, t^y, t^w, t^h)$ and the ground-truth transform.\n",
    "\n",
    "\n",
    "Run the following to check your implementation. You should see errors on the order of 1e-7 or less.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 115901,
     "status": "ok",
     "timestamp": 1604338994441,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "beEUhlCHtFAN",
    "new_sheet": false,
    "outputId": "b209a21a-6571-46c5-e36e-5873d4802093",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import GenerateProposal\n",
    "\n",
    "print('-' * 80)\n",
    "print('Sanity check for YOLO.')\n",
    "yolo_offset_list = torch.cat([torch.ones_like(anc_list[:, :, :, :, 0:2]).fill_(.5), torch.ones_like(anc_list[:, :, :, :, 2:4])], dim=-1)\n",
    "yolo_proposal_list = GenerateProposal(anc_list, yolo_offset_list, 'YOLO') # no scaling\n",
    "\n",
    "print('1. Center moved by ~0.5 cell')\n",
    "print('rel error: ', rel_error(anc_list[0, 0, 0, :, 0:2] + (anc_list[0, 0, 0, :, 2:4] - anc_list[0, 0, 0, :, 0:2])/2.0 + 0.5, \\\n",
    "                               (yolo_proposal_list[0, 0, 0, :, 0:2] + (yolo_proposal_list[0, 0, 0, :, 2:4] - yolo_proposal_list[0, 0, 0, :, 0:2]) / 2.0)))\n",
    "\n",
    "print('2. w/h changed by e')\n",
    "print('rel error: ', rel_error((anc_list[0, 0, 0, :, 2:4] - anc_list[0, 0, 0, :, 0:2]) * torch.exp(torch.ones_like(anc_list[0, 0, 0, :, 0:2])), \\\n",
    "      (yolo_proposal_list[0, 0, 0, :, 2:4] - yolo_proposal_list[0, 0, 0, :, 0:2])))\n",
    "\n",
    "\n",
    "print('-' * 80)\n",
    "print('Sanity check for FasterRCNN.')\n",
    "rcnn_offset_list = torch.ones_like(anc_list)\n",
    "rcnn_proposal_list = GenerateProposal(anc_list, rcnn_offset_list, 'FasterRCNN')\n",
    "\n",
    "print('1. x/y shifted by wh')\n",
    "print('rel error: ', rel_error(anc_list[0, 0, 0, :, 0:2] + (anc_list[0, 0, 0, :, 2:4] - anc_list[0, 0, 0, :, 0:2]) * 3.0 /2.0, \\\n",
    "      (rcnn_proposal_list[0, 0, 0, :, 0:2] + (rcnn_proposal_list[0, 0, 0, :, 2:4] - rcnn_proposal_list[0, 0, 0, :, 0:2]) / 2.0)))\n",
    "\n",
    "print('2. w/h should changed by e')\n",
    "print('rel error: ', rel_error((anc_list[0, 0, 0, :, 2:4] - anc_list[0, 0, 0, :, 0:2]) * torch.exp(torch.ones_like(anc_list[0, 0, 0, :, 0:2])), \\\n",
    "      (rcnn_proposal_list[0, 0, 0, :, 2:4] - rcnn_proposal_list[0, 0, 0, :, 0:2])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "gYzP-8tGo6bp",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As an additional sanity check, we visualize an anchor (in red) and the corresponding proposal (in green) that results from applying the transform $(0.5, 0.5, 0, 0)$. The proposal should shift down and to the right (when using the YOLO formulation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 117701,
     "status": "ok",
     "timestamp": 1604338996258,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "YQZH2b7fhS49",
    "new_sheet": false,
    "outputId": "a148fe5d-ebc3-4a29-f13f-8aa542fc1e31",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "yolo_offset_list = torch.cat([torch.ones_like(anc_list[:, :, :, :, 0:2]).fill_(.5), torch.zeros_like(anc_list[:, :, :, :, 2:4])], dim=-1)\n",
    "yolo_proposal_list = GenerateProposal(anc_list, yolo_offset_list, 'YOLO')\n",
    "\n",
    "for img, anc, prop in zip(img_list, coord_trans(anc_list[:, 0:1, 3:4, 3:4, :], w_list, h_list), \\\n",
    "                          coord_trans(yolo_proposal_list[:, 0:1, 3:4, 3:4, :], w_list, h_list)):\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4), prop.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "WvR1zYcMpdYU",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next we visualize the effect of applying the transform $(0, 0, 1, 1)$ (with the YOLO formula) to the same anchor box (in red). Now the proposal (in green) should have the same center as the anchor, but the proposal should be larger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 117687,
     "status": "ok",
     "timestamp": 1604338996259,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "3LtrWZCeyYVS",
    "new_sheet": false,
    "outputId": "7d01c459-0e65-4787-9a4d-ef643500217c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "yolo_offset_list = torch.cat([torch.zeros_like(anc_list[:, :, :, :, 0:2]), torch.ones_like(anc_list[:, :, :, :, 2:4]).fill_(1.)], dim=-1)\n",
    "yolo_proposal_list = GenerateProposal(anc_list, yolo_offset_list, 'YOLO')\n",
    "\n",
    "for img, anc, prop in zip(img_list, coord_trans(anc_list[:, 0:1, 3:4, 3:4, :], w_list, h_list), \\\n",
    "                          coord_trans(yolo_proposal_list[:, 0:1, 3:4, 3:4, :], w_list, h_list)):\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4), prop.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "d7WJQkOkp6E5",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next to sanity-check our implementation of the Faster R-CNN transformation formulas, we visualize the effect of applying the transform $(1, 1, 0, 0)$ to the same anchor (in red). The proposal (in green) should shift down and to the right by an amount equal to the size of the anchor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 120498,
     "status": "ok",
     "timestamp": 1604338999087,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "Fk6FWCcYh1hZ",
    "new_sheet": false,
    "outputId": "f97f0985-a671-4655-fbd8-9872829e8e71",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualization (shift by wh, Faster R-CNN)\n",
    "# anchors in red and proposals in green\n",
    "rcnn_offset_list = torch.cat([torch.ones_like(anc_list[:, :, :, :, 0:2]), torch.zeros_like(anc_list[:, :, :, :, 2:4])], dim=-1)\n",
    "rcnn_proposal_list = GenerateProposal(anc_list, rcnn_offset_list, 'FasterRCNN')\n",
    "\n",
    "for img, anc, prop in zip(img_list, coord_trans(anc_list[:, 0:1, 3:4, 3:4, :], w_list, h_list), \\\n",
    "                          coord_trans(rcnn_proposal_list[:, 0:1, 3:4, 3:4, :], w_list, h_list)):\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4), prop.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Ex-G6inJqOgv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We further check our implementation of the Faster R-CNN transformation formula, and visualize the effect of applying the transformation $(0, 0, 1, 1)$ to the same anchor. Now the proposal (in green) should have the same center as the anchor (in red), but the proposal should be larger by a factor of $e$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 120484,
     "status": "ok",
     "timestamp": 1604338999089,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "rzDVtYz1hjVR",
    "new_sheet": false,
    "outputId": "161c6dda-d574-4bde-bc0c-d8db7367e150",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualization (no shift and then scale by e, Faster R-CNN)\n",
    "# anchors in red and proposals in green\n",
    "rcnn_offset_list = torch.cat([torch.zeros_like(anc_list[:, :, :, :, 0:2]), torch.ones_like(anc_list[:, :, :, :, 2:4]).fill_(1)], dim=-1)\n",
    "rcnn_proposal_list = GenerateProposal(anc_list, rcnn_offset_list, 'FasterRCNN')\n",
    "\n",
    "for img, anc, prop in zip(img_list, coord_trans(anc_list[:, 0:1, 3:4, 3:4, :], w_list, h_list), \\\n",
    "                          coord_trans(rcnn_proposal_list[:, 0:1, 3:4, 3:4, :], w_list, h_list)):\n",
    "  detection_visualizer(img, idx_to_class, anc.reshape(-1, 4), prop.reshape(-1, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "_lBcOWGWtNka",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Prediction Networks\n",
    "This module outputs the prediction scores (see figure below). We have provided the code to determine activated/negative anchors for you, though you need to implement the IoU function for it to work. You will also need to compute the loss function. The loss function consists of three parts, confidence score regression, bounding box offsets regression, and object classication.\n",
    "\n",
    "![pred_scores](https://miro.medium.com/max/1055/1*YG6heD55fEmZeUKRSlsqlA.png)\n",
    "\n",
    "\n",
    "Image credit: [towardsdatascience](https://towardsdatascience.com/yolov1-you-only-look-once-object-detection-e1f3ffec8a89). In this example, number of anchor shapes is $A=2$ while we have $A=9$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Q8HqHqojtXhE",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Intersection Over Union (IoU)\n",
    "The definition of IoU and instructions on how to compute IoU can be found in the lecture slides (p35-39): https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture15.pdf\n",
    "\n",
    "Run the following to check your implementation. You should see errors on the order of 1e-8 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 120469,
     "status": "ok",
     "timestamp": 1604338999089,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "IRXHJXFIbbDs",
    "new_sheet": false,
    "outputId": "21539fd6-e162-4b7a-f542-5ef5e18b4d2e",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import IoU\n",
    "\n",
    "# simple sanity check\n",
    "width = torch.tensor([35, 35], dtype=torch.float32, device='cuda')\n",
    "heigh = torch.tensor([40, 40], dtype=torch.float32, device='cuda')\n",
    "sample_bbox = torch.tensor([[[1,1,11,11,0], [20,20,30,30,0]]], dtype=torch.float32, device='cuda')\n",
    "sample_proposals = torch.tensor([[[[[5,5,15,15], [27,27,37,37]]]]], dtype=torch.float32, device='cuda')\n",
    "\n",
    "\n",
    "result = IoU(sample_proposals, sample_bbox)\n",
    "\n",
    "# check 1\n",
    "expected_result = torch.tensor([[[0.21951219, 0.00000000],\n",
    "                                 [0.00000000, 0.04712042]]], dtype=torch.float32, device='cuda')\n",
    "print('simple iou_mat error: ', rel_error(expected_result, result))\n",
    "\n",
    "# check 2\n",
    "iou_mat = IoU(anc_list, resized_box_list)\n",
    "expected_iou_mat = torch.tensor([0.11666405, 0.15146968, 0.02956639], dtype=torch.float32, device='cuda')\n",
    "print('iou_mat error: ', rel_error(expected_iou_mat, iou_mat[:, :, 0].mean(1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "WNSwO-wDwzoQ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Activated (positive) and negative anchors\n",
    "During training we need to match the ground-truth boxes against the anchors to determine the classification labels for the anchors -- which anchors should be classified as containing an object and which should be classified as background? We have written this part for you.\n",
    "\n",
    "Read and digest the input/output definition carefully. You are highly recommended to read through the code as later parts rely heavily on this function.\n",
    "\n",
    "The function `ReferenceOnActivatedAnchors` is implemented in `a5_helper.py`.\n",
    "\n",
    "Run the following to check your implementation. You should see errors on the order of 1e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 120455,
     "status": "ok",
     "timestamp": 1604338999090,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "fK_USCuaXSzh",
    "new_sheet": false,
    "outputId": "9548c246-391e-4a0f-a366-a875d925d140",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# sanity check on YOLO (the one on Faster R-CNN is in A5-2)\n",
    "activated_anc_ind, negative_anc_ind, GT_conf_scores, GT_offsets, GT_class, \\\n",
    "  activated_anc_coord, negative_anc_coord = ReferenceOnActivatedAnchors(anc_list, resized_box_list, grid_list, iou_mat, neg_thresh=0.2, method='YOLO')\n",
    "\n",
    "expected_GT_conf_scores = torch.tensor([0.74538743, 0.72793430, 0.76044953, 0.37116671], device='cuda', dtype=torch.float32)\n",
    "expected_GT_offsets = torch.tensor([[ 0.04900002,  0.35735703, -0.09431065,  0.19244696],\n",
    "                                    [-0.14700007,  0.37299442, -0.00250307,  0.25213102],\n",
    "                                    [-0.40600014,  0.09625626,  0.20863886, -0.07974572],\n",
    "                                    [ 0.15399981, -0.42933345, -0.03459148, -0.86750042]], device='cuda', dtype=torch.float32)\n",
    "expected_GT_class = torch.tensor([ 6,  7, 19,  6], device='cuda', dtype=torch.int64)\n",
    "print('conf scores error: ', rel_error(GT_conf_scores, expected_GT_conf_scores))\n",
    "print('offsets error: ', rel_error(GT_offsets, expected_GT_offsets))\n",
    "print('class prob error: ', rel_error(GT_class, expected_GT_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "YsPZw2mTutWP",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can sanity check this function by visualizing ground-truth boxes (in red) along with positive / negative anchors (in green)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 121581,
     "status": "ok",
     "timestamp": 1604339000232,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "2TuJNoCvUuqc",
    "new_sheet": false,
    "outputId": "f7bbed8c-ceed-48ae-b41b-877f3465f8ba",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize activated and negative anchors\n",
    "anc_per_img = torch.prod(torch.tensor(anc_list.shape[1:-1]))\n",
    "\n",
    "print('*'*80)\n",
    "print('Activated (positive) anchors:')\n",
    "for img, bbox, idx in zip(img_list, box_list, torch.arange(box_list.shape[0])):\n",
    "  anc_ind_in_img = (activated_anc_ind >= idx * anc_per_img) & (activated_anc_ind < (idx+1) * anc_per_img)\n",
    "  print('{} activated anchors!'.format(torch.sum(anc_ind_in_img)))\n",
    "  detection_visualizer(img, idx_to_class, bbox[:, :4], coord_trans(activated_anc_coord[anc_ind_in_img], w_list[idx], h_list[idx]))\n",
    "\n",
    "print('*'*80)\n",
    "print('Negative anchors:')\n",
    "for img, bbox, idx in zip(img_list, box_list, torch.arange(box_list.shape[0])):\n",
    "  anc_ind_in_img = (negative_anc_ind >= idx * anc_per_img) & (negative_anc_ind < (idx+1) * anc_per_img)\n",
    "  print('{} negative anchors!'.format(torch.sum(anc_ind_in_img)))\n",
    "  detection_visualizer(img, idx_to_class, bbox[:, :4], coord_trans(negative_anc_coord[anc_ind_in_img], w_list[idx], h_list[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "XW_Zek3_dgfF",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Prediction Network\n",
    "The prediction network inputs the features from the backbone network, and outputs the classification scores and transformations for each anchor.\n",
    "\n",
    "For each position in the 7x7 grid of features from the backbone, the prediction network outputs `C` numbers to be interpreted as classification scores over the `C` object categories for the anchors at that position.\n",
    "\n",
    "In addition, for each of the `A` anchors at each position, the prediction network outputs a transformation (4 numbers, to convert the anchor box into a region proposal) and a confidence score (where large positive values indicate high probability that the anchor contains an object, and large negative values indicate low probability that the anchor contains an object).\n",
    "\n",
    "Collecting all of these outputs, we see that for each position in the 7x7 grid of features we need to output a total of `5A+C` numbers, so the prediction network receives an input tensor of shape `(B, 1280, 7, 7)` and produces an output tensor of shape `(B, 5A+C, 7, 7)`. We can achieve this with two `1x1` convolution layers operating on the input tensor, where the number of filters in the second layer is `5A+C`.\n",
    "\n",
    "During training, we do not apply the loss on the full set of anchor boxes for the image; instead we designate a subset of anchors as positive and negative by matching them with ground-truth boxes as in the function above. The Prediction network is also responsible for picking out the outputs corresponding to the positive and negative anchors.\n",
    "\n",
    "Implement `PredictionNetwork` in `single_stage_detector.py` and run the following to check your implementation. You should see errors on the order of 1e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 121566,
     "status": "ok",
     "timestamp": 1604339000233,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "4lgXGvq7JOmS",
    "new_sheet": false,
    "outputId": "0415bf50-fe7c-4de3-fe14-27ac9b2b917f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import PredictionNetwork\n",
    "\n",
    "# sanity check\n",
    "reset_seed(0)\n",
    "pred_network = PredictionNetwork(1280, drop_ratio=0).to(dtype=torch.float32, device='cuda')\n",
    "features = torch.linspace(-10., 10., steps=3*1280*7*7, dtype=torch.float32, device='cuda').view(3, 1280, 7, 7)\n",
    "pos_anchor_idx = torch.tensor([122, 605, 871, 955], dtype=torch.int64, device='cuda')\n",
    "neg_anchor_idx = torch.tensor([1048, 1292, 1124, 1092], dtype=torch.int64, device='cuda')\n",
    "conf_scores, offsets, class_prob = pred_network(features, pos_anchor_idx, neg_anchor_idx)\n",
    "\n",
    "expected_conf_scores = torch.tensor([[0.85080749], [0.55230302], [0.45239496], [0.51819414],\n",
    "                                     [0.38467780], [0.82627463], [0.53837817], [0.538446366]], dtype=torch.float32, device='cuda')\n",
    "expected_offsets = torch.tensor([[-0.16344640,  0.45598251, -1.56108809, -1.96177566],\n",
    "                                 [ 0.09087485, -0.11321104,  0.46109992,  0.13557276],\n",
    "                                 [ 0.14098871, -0.04003078, -0.14689390,  0.01480492],\n",
    "                                 [-0.22611487,  0.14513946, -1.26124716,  0.58137202]], dtype=torch.float32, device='cuda')\n",
    "expected_class_prob = torch.tensor([0.88649291, 1.08752346, 0.90241265, 1.28021181, 0.77324629,\n",
    "                                    -0.93609941, 1.44269419, -1.01072836, 0.09888625, -0.09179375,\n",
    "                                    -0.48814785, -1.14749694, 0.24533349, -0.19522685, 0.21445289,\n",
    "                                    0.91779679, 0.97305167, 1.01984429, -0.11932681, 0.34998628], dtype=torch.float32, device='cuda')\n",
    "\n",
    "print('conf scores error: ', rel_error(conf_scores, expected_conf_scores))\n",
    "print('offsets error: ', rel_error(offsets, expected_offsets))\n",
    "print('class prob error: ', rel_error(class_prob.mean(0), expected_class_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ah05Gd6BOKG2",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Loss Function\n",
    "The confidence score regression loss is for both activated/negative anchors while the bounding box regression loss and the object classification loss are for activated anchors only. These are implemented for you. You can see the functions `ConfScoreRegression` and `BboxRegression` in `single_stage_detector.py`, and `ObjectClassification` in `a5_helper.py`.\n",
    "\n",
    "Run the following to check your implementation. You should see errors on the order of 1e-8 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 121548,
     "status": "ok",
     "timestamp": 1604339000233,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "B0iG-DAUOQ56",
    "new_sheet": false,
    "outputId": "d2aee15f-1a4e-4f1f-827f-a2f1aae9990a",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# sanity check\n",
    "from single_stage_detector import ConfScoreRegression, BboxRegression\n",
    "\n",
    "\n",
    "conf_loss = ConfScoreRegression(conf_scores, GT_conf_scores)\n",
    "reg_loss = BboxRegression(offsets, GT_offsets)\n",
    "cls_loss = ObjectClassification(class_prob, GT_class, w_list.shape[0], anc_per_img, activated_anc_ind)\n",
    "print('conf loss: {:.4f}, reg loss: {:.4f}, cls loss: {:.4f}'.format(conf_loss, reg_loss, cls_loss))\n",
    "\n",
    "loss_all = torch.tensor([conf_loss.data, reg_loss.data, cls_loss.data], device='cuda', dtype=torch.float32)\n",
    "expected_loss = torch.tensor([0.27064770460128784, 2.9750006198883057, 2.3919472694396973], device='cuda', dtype=torch.float32)\n",
    "\n",
    "print('loss error: ', rel_error(loss_all, expected_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "MIt5AxlAxwKz",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Train an object detector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "7yCYzKIxx2qB",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Object detection module\n",
    "\n",
    "We will now combine everything into the `SingleStageDetector` class in `single_stage_detector.py`. Implement the `forward` function of our detector. This implements the training-time forward pass: it receives the input images and the ground-truth bounding boxes, and returns the total loss for the minibatch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "D-9nFPtLyDE_",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Overfit small data\n",
    "To make sure that everything is working as expected, we can try to overfit the detector to a small subset of data.\n",
    "\n",
    "After 200 epochs of training you should see a total loss of around or less than 0.3.\n",
    "\n",
    "\n",
    "We also implemented the `DetectionSolver` object which runs the training loop to train an single stage detector. You can inspect the DetectionSolver implementation in `single_stage_detector.py`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 143811,
     "status": "ok",
     "timestamp": 1604339022516,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "eNKLRL7HyHO0",
    "new_sheet": false,
    "outputId": "961a6738-263a-40a8-cad1-1bd701d2eff7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import DetectionSolver, SingleStageDetector\n",
    "\n",
    "# monitor the training loss\n",
    "reset_seed(0)\n",
    "num_sample = 10\n",
    "small_dataset = torch.utils.data.Subset(train_dataset, torch.linspace(0, len(train_dataset)-1, steps=num_sample).long())\n",
    "small_train_loader = pascal_voc2007_loader(small_dataset, 10) # a new loader\n",
    "\n",
    "for lr in [1e-2]:\n",
    "  print('lr: ', lr)\n",
    "  detector = SingleStageDetector()\n",
    "  DetectionSolver(detector, small_train_loader, learning_rate=lr, num_epochs=200, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "CuSBfcGWyHlD",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Train a net\n",
    "\n",
    "Now that we are confident that the training code is working properly, let's train the network on more data and for longer. We will train for 50 epochs; this should take about 35 minutes on a K80 GPU. You should see a total loss around or less than 0.27.\n",
    "\n",
    "For initial debugging, you may want to train for a smaller number of epochs; training for 5 epochs, you can expect a loss of around 1.0. \n",
    "\n",
    "Note that real object detection systems typically train for 12-24 hours, distribute training over multiple GPUs, and use much faster GPUs. As such our result will be far from the state of the art, but it should give some reasonable results!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1305188,
     "status": "ok",
     "timestamp": 1604344026014,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "Aipf7-XQyJ28",
    "new_sheet": false,
    "outputId": "2a180cca-46b0-4951-87ca-e903646e20d5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# monitor the training loss\n",
    "train_loader = pascal_voc2007_loader(train_dataset, 100)\n",
    "\n",
    "lr = 5e-2\n",
    "num_epochs = 50\n",
    "yolo_detector = SingleStageDetector().to(dtype=torch.float32, device='cuda')\n",
    "DetectionSolver(yolo_detector, train_loader, learning_rate=lr, num_epochs=num_epochs, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "hmk6rwb-C5al",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After you've trained your model, save the weights for submission. Feel free to comment some of those lines to load earlier checkpoints. We evaluate the detector after loading as a sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 19786,
     "status": "ok",
     "timestamp": 1604344057169,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "j_K7nL8eviXV",
    "new_sheet": false,
    "outputId": "800e2145-0946-4829-9137-590149d2e17f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "weights_path = os.path.join(GOOGLE_DRIVE_PATH, \"yolo_detector.pt\")\n",
    "torch.save(yolo_detector.state_dict(), weights_path)\n",
    "\n",
    "yolo_detector = SingleStageDetector().to(dtype=torch.float32, device='cuda')\n",
    "yolo_detector.load_state_dict(torch.load(weights_path))\n",
    "DetectionEvaluater(yolo_detector, train_loader, dtype=torch.float32, device='cuda')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "xzU71hc_y9Ij",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Use an object detector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "e42TAEcpjeKW",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Non-Maximum Suppression (NMS)\n",
    "The definition of NMS and instructions on how to compute NMS can be found in the lecture slides (p40-44):\n",
    "https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture15.pdf\n",
    "\n",
    "Implement the `nms` function in `single_stage_detector.py`. We then compare your implementation of NMS with the implementation in torchvision. Most likely, your implementation will be faster on CPU than on CUDA, and the torchvision implementation will likely be much faster than yours. This is expected, but your implementation should produce the same outputs as the torchvision version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1474,
     "status": "ok",
     "timestamp": 1604344103435,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "wqXkUdvdHh-U",
    "new_sheet": false,
    "outputId": "5ece168a-25f2-4b8f-d31a-5ad281b8c772",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import nms\n",
    "\n",
    "reset_seed(0)\n",
    "boxes = (100. * torch.rand(5000, 4)).round()\n",
    "boxes[:,2] = boxes[:,2] + boxes[:,0] + 1.\n",
    "boxes[:,3] = boxes[:,3] + boxes[:,1] + 1.\n",
    "scores = torch.randn(5000)\n",
    "\n",
    "names = ['your_cpu', 'torchvision_cpu', 'torchvision_cuda']\n",
    "iou_thresholds = [0.3, 0.5, 0.7]\n",
    "elapsed = dict(zip(names, [0.]*len(names)))\n",
    "intersects = dict(zip(names[1:], [0.]*(len(names)-1)))\n",
    "\n",
    "for iou_threshold in iou_thresholds:\n",
    "  tic = time.time()\n",
    "  my_keep = nms(boxes, scores, iou_threshold)\n",
    "  elapsed['your_cpu'] += time.time() - tic\n",
    " \n",
    "  tic = time.time()\n",
    "  tv_keep = torchvision.ops.nms(boxes, scores, iou_threshold)\n",
    "  elapsed['torchvision_cpu'] += time.time() - tic\n",
    "  intersect = len(set(tv_keep.tolist()).intersection(my_keep.tolist())) / len(tv_keep)\n",
    "  intersects['torchvision_cpu'] += intersect\n",
    "\n",
    "  tic = time.time()\n",
    "  tv_cuda_keep = torchvision.ops.nms(boxes.cuda(), scores.cuda(), iou_threshold).to(my_keep.device)\n",
    "  torch.cuda.synchronize()\n",
    "  elapsed['torchvision_cuda'] += time.time() - tic\n",
    "  intersect = len(set(tv_cuda_keep.tolist()).intersection(my_keep.tolist())) / len(tv_cuda_keep)\n",
    "  intersects['torchvision_cuda'] += intersect\n",
    "\n",
    "for key in intersects:\n",
    "  intersects[key] /= len(iou_thresholds)\n",
    "\n",
    "# You should see < 1% difference\n",
    "print('Testing NMS:')\n",
    "print('Your        CPU  implementation: %fs' % elapsed['your_cpu'])\n",
    "print('torchvision CPU  implementation: %fs' % elapsed['torchvision_cpu'])\n",
    "print('torchvision CUDA implementation: %fs' % elapsed['torchvision_cuda'])\n",
    "print('Speedup CPU : %fx' % (elapsed['your_cpu'] / elapsed['torchvision_cpu']))\n",
    "print('Speedup CUDA: %fx' % (elapsed['your_cpu'] / elapsed['torchvision_cuda']))\n",
    "print('Difference CPU : ', 1. - intersects['torchvision_cpu']) # in the order of 1e-3 or less\n",
    "print('Difference CUDA: ', 1. - intersects['torchvision_cuda']) # in the order of 1e-3 or less"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "9JSTPMsqzEnr",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Inference\n",
    "\n",
    "Now, implement the inference part of module `SingleStageDetector`. Once you're done, test your implementation using the overfitted model weights. \n",
    "\n",
    "Visualize the output from the overfitted model on the small dataset by running the code box below, the bounding boxes should be really accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "output_embedded_package_id": "1b1NLbE8JiqblTfMt75oJA_2rmoCf_scm"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 4559,
     "status": "ok",
     "timestamp": 1604344112451,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "gp_Hmt-Km5bl",
    "new_sheet": false,
    "outputId": "d2ea2123-0384-498b-c0c0-0829ae4dba65",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "DetectionInference(detector, small_train_loader, small_dataset, idx_to_class, thresh=0.8, device='cuda', dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ifdEPmd9EMCP",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Inference - train a net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "output_embedded_package_id": "1UIuwow45uj610xiJjnyji1Ivfa2vjSpY"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 4601,
     "status": "ok",
     "timestamp": 1604344120527,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "J7ArGiLTnHta",
    "new_sheet": false,
    "outputId": "7f7740ce-1c66-48f2-a546-50db154438a3",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize the same output from the model trained on the entire training set\n",
    "# some bounding boxes might not make sense\n",
    "DetectionInference(yolo_detector, small_train_loader, small_dataset, idx_to_class, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ETU6ev7aydIY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Evaluation\n",
    "Compute mean Average Precision (mAP). Introduction on mAP see lecture slides (p46-57): https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture15.pdf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "1fGptrealquF",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Run the following to evaluate your detector on the PASCAL VOC validation set. You should see mAP at around 11% or above.\n",
    "\n",
    "The state of the art on this dataset is >80% mAP! To achieve these results we would need to use a much bigger network, and train with more data and for much longer, but that is beyond the scope of this assigment.\n",
    "\n",
    "(Optional) If you train the model longer (e.g., 100 epochs), you should see a better mAP. But make sure you revert the code back for grading purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 41965,
     "status": "ok",
     "timestamp": 1604344169809,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "FvDb7uwqyhAK",
    "new_sheet": false,
    "outputId": "e8024dfc-1131-4a3d-a629-3f8997bfb7de",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "DetectionInference(yolo_detector, val_loader, val_dataset, idx_to_class, output_dir='mAP/input', thresh=0.8, nms_thresh=0.3, device='cuda', dtype=torch.float32)\n",
    "!cd mAP && python main.py"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "single_stage_detector_yolo.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "39f2c90a30784ccd95c885861a8ae353": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "69460e1899694f06a47c55f4fd2cab38": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cb76667a3a5f4e9a8100e2407b2bdae8",
      "placeholder": "​",
      "style": "IPY_MODEL_8d6a6ba882824d40a258d0eca1b70335",
      "value": " 13.6M/13.6M [00:00&lt;00:00, 17.5MB/s]"
     }
    },
    "72cee37f094d4bf98286cf609591a065": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_fcb7f0f5206242f9abef713c3fcf3d59",
       "IPY_MODEL_c6de2c2b7f6d43f3bc5db37780aa6bc9"
      ],
      "layout": "IPY_MODEL_af39fda1643342619373531854ec5a6a"
     }
    },
    "7698f88da6ed494886f88d808869555c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "876bc727f1a341868e3ef42ba2763408": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8d6a6ba882824d40a258d0eca1b70335": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "98ee12dbedf24de884be7c596bf4228a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9afd2271777144eea2b996de9747975c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ad0732df20c546ad99c079fb55fd5110": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "af39fda1643342619373531854ec5a6a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b7417b5be2ee4915be4dc010d170ce7c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9afd2271777144eea2b996de9747975c",
      "max": 14212972,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f45f1f174a9e479aa6f6e1fa61527af1",
      "value": 14212972
     }
    },
    "bbdfd4e4a86a48fb86b2063510dd53d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b7417b5be2ee4915be4dc010d170ce7c",
       "IPY_MODEL_69460e1899694f06a47c55f4fd2cab38"
      ],
      "layout": "IPY_MODEL_876bc727f1a341868e3ef42ba2763408"
     }
    },
    "c6de2c2b7f6d43f3bc5db37780aa6bc9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_98ee12dbedf24de884be7c596bf4228a",
      "placeholder": "​",
      "style": "IPY_MODEL_ad0732df20c546ad99c079fb55fd5110",
      "value": " 460038144/? [01:10&lt;00:00, 9269400.25it/s]"
     }
    },
    "cb76667a3a5f4e9a8100e2407b2bdae8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f45f1f174a9e479aa6f6e1fa61527af1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "fcb7f0f5206242f9abef713c3fcf3d59": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7698f88da6ed494886f88d808869555c",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_39f2c90a30784ccd95c885861a8ae353",
      "value": 1
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
